Web Scraping Whoopsies

Manu Alcalá + Jameson Carter

2023-10-18

What is web scraping?

Broadly, it is the process of automatically pulling data from websites by reading their underlying code. Doing this gets complicated fast:

Static websites contain information in HTML code which does not change when you, the user, interact with it.
Dynamic websites have information in HTML code which does change as you interact with it.
Static vs. Dynamic websites require different code packages to scrape.
- For example, selenium is a common dynamic web scraping library and scrapy is a common static web scraping library.
- This is because a static scraper simply has to understand what it is looking for in the HTML. A dynamic scraper needs to simulate a human interacting with the site.

Generally, web scraping is the process of automatically reading the content of websites, in our case to get data. Within a website’s underlying code, there might be something that you want to access. For example, a large table published on a website with no download button. But getting that hypothetical table depends on how the website is set up. Static sites are relatively easy to scrape because all of the information is available to you through the code which built the site, at all times. Dynamic sites work a little differently: they require you, the user, to click on buttons and interact with the site.

Because of this, the tools we use differ. Static scrapers usually pull the sites’ code directly and then parse that code to extract relevant information. Dynamic scrapers simulate a human being by clicking on the right series of buttons, to trick the site into providing its data, which can then be parsed. You will hear us mention both scrapy and selenium today. Selenium is used for dynamic sites and scrapy is used for static sites .

When is it worth it?

Other data collection efforts are impossible or prone to human error
- Creating reproducible and reliable processes eases collaboration / quality assurance
Getting new data releases quickly is valuable
Time spent coding < time spent obtaining data in other ways + time spent on quality assurance

As you might have guessed, this takes some time. Before getting into it, you should consider whether other methods of data collection are error-prone or so daunting that you could never justify doing it by hand. You have emailed the owner of the data to no avail and you fear that manually copying 1000 entries could be subject to error (or a waste of time). Also, it is easier to collaborate with code-based processes, explaining how you hand-generated tables can sometimes feel like more trouble than its worth!

Additionally, consider whether getting new data quickly is valuable. For example, if a government site releases data regularly you may want to have a process ready to extract those data. Web scraping is great for this.

Finally, note that when planning these things it might seem like a bunch of code on the front-end. But if the time spent coding is less than time spent obtaining data in other ways PLUS time spent on quality assurance of those methods, it could be well worth it! Validating processes is easier with code.

What makes it hard?

It is difficult to predict how much time a web scraping task will take.
Sites might change, introducing need to update.
- Sites might be removed, stop being maintained, or completely overhauled.
Site maintainers may not be okay with data being scraped. Quick plug for the TECH team’s Automated Data Guidelines.

Example 1. Scraping Medicaid enrollment data

Why Medicaid enrollment data?

Since Spring 2023, states have been dis-enrolling Medicaid beneficiaries who no longer qualify since the Public Health Emergency was ended.

Why are the data interesting?

In anticipation of “the great unwinding,” many states implemented policy changes to smooth the transition.

To understand the success of these policies, we wanted time-series enrollment data for all 50 states… from a Medicaid data system that is largely decentralized.

Unreadable PDFs abound!

A sigh of relief…

Why page through PDFs when another organization’s RAs can do it for you?

1. Identify this is a scrape-able dynamic page

One URL with data you can only get by clicking each option!

2. Confirm HTML actually contains the data

3. Code for 30 hours!

4. Bask in the glow of automated scraping

Whenever new data were released in the following 2 months, I re-ran the code and got a well-formatted excel file as output.

Little did I know, trouble was coming

What happened?

2 months later, KFF stopped updating the dashboard and changed how existing data was reported on graphs.

Example 2. Scraping Course Catalogs

Why course catalog data?

States are increasingly interested work-based learning (WBL) as an important strategy for helping students prepare for and access good jobs, but measurement has been limited
To understand the prevalence and types of WBL, we wanted course-level data from community colleges across the country

Not all catalogs are the same

An example of course descriptions listed under department pages

And an example containing links to course catalogs in .pdf format

Web crawling adventures

Scrapy is a web scraping and web crawling framework to extract structured data from websites
It uses spiders, which are self-contained web crawlers that follow links according to your instructions

Given the complexity of our problem, we needed a really powerful tool that could scale well. I got really excited about Scrapy since it’s a framework designed specifically for scraping data at scale. In addition to extracting and storing web data, it uses “Spiders” which are self-contained web crawlers that can follow links and navigate through websites according to rules you specify, making it easier to scrape a large amount of data.

In addition, Scrapy offers game changing features for more complex projects where you might need to manage cookies, write to a database, be careful about not overwhelming servers, and respecting mantainers rules for what data you can scrape.

So I decided it was worth learning and wanted to create a crawler that could start from a home page and follow links that contained keywords related to course catalogs. I powered through Scrapy’s steep learning curve but after much pain and confusion, I finally got my spider to work on a small sample of schools!

Scrapy go brrrrr…

Whoopsie…

We were not ready. After a lot of hard work and many many hours refining this approach, our data was a hot mess: Our spider failed to collect any data for around 1/3 of schools, which wasn’t great but given that no data on WBL exists, it still could’ve been useful if the remaining data were usable. But alas, most of our data had all sorts of problems that made us nervous about using it.

Often multiple courses are listed on a page and we couldn’t single them out courses without directly targeting the HTML. Our data also contained text from irrelevent links and we weren’t able to develop rules that would correctly identify courses consistently. Another major problem was that we couldn’t even verify whether we had found all course descriptions for a particular school.

So evidently, there were many many challenges with scraping at this scale that we didn’t consider in advance and we decided to narrow our focus to a single state instead.

A new direction

Selenium and BeautifulSoup to the rescue!

Concluding remarks

Core questions to explore before you start

Availability of data

Are the data available through other routes?
Are the data produced by an organization that is invested in the problem long-term?

Frequency of scraping

Will I need to scrape the data multiple times?
What is the risk that the item scraped from the site will be changed?

Time-value tradeoffs

Is the time spent coding worth the payoff?
Will collecting data automatically save time on quality assurance?

Questions?

The remainder of the time is reserved for group discussion!

Thank you!

Please contact Manu Alcalá or Jameson Carter if you would like to discuss either of these projects or scope whether a use-case is reasonable.